[Job - 20712] Mid-Level Data Engineer, Colombia

As a Mid-Level Data Engineer at CI&T, you'll be responsible for developing and optimizing scalable data pipelines using Python and AWS services, managing ETL/ELT workflows with tools like Airflow and Glue, and ensuring data integrity and availability. You'll also work to improve storage efficiency and query performance for large datasets, implement logging and monitoring for pipeline reliability, and collaborate with cross-functional teams to enhance decision-making through quality data.

To qualify for the Mid-Level Data Engineer role at CI&T, candidates should have relevant experience in Data Engineering or a similar role, strong Python proficiency for data processing, and solid experience with AWS and basic SQL. Hands-on experience with ETL orchestration using Apache Airflow is also essential, along with familiarity in CI/CD pipelines and version control systems like Git.

While not mandatory, certain qualifications can boost your application for the Mid-Level Data Engineer position at CI&T. These include experience with data lake architectures, knowledge of data governance and cost management best practices, and prior experience working with IoT data pipelines. Highlighting these will give you an edge in the application process.

CI&T offers a robust benefits package for Mid-Level Data Engineers, including premium healthcare, meal vouchers, maternity and parental leaves, mobile services subsidy, sick pay, life insurance, and professional development opportunities through CI&T University. You will also enjoy Colombian holidays and paid vacation days, ensuring a balanced work-life environment.

At CI&T, we believe that collaboration is our superpower and diversity unites us. We are committed to fostering a diverse, inclusive, and safe work environment for all our employees, including Data Engineers. We encourage applications from diverse and underrepresented groups, and we prioritize creating a workplace where everyone's identities and life experiences are valued and respected.

To ensure data integrity in your pipelines, you can implement checks such as validation rules, duplication checks, and reconciliation processes. You might also discuss using tools like Apache Airflow for orchestrating tasks to minimize errors and maintain data consistency.

Share specific AWS services you have worked with, such as AWS S3 for storage, AWS Lambda for serverless computing, or AWS Glue for ETL processes. Explain how these tools have helped you build efficient and scalable data solutions.

Provide a specific example where you faced a challenge, such as a failed job or data discrepancies. Describe the steps you took to diagnose the issue and the solution you implemented, emphasizing your problem-solving skills and persistence.

Discuss various strategies to optimize performance, such as partitioning data, using efficient data formats, indexing, and caching strategies. Provide examples of how you've successfully implemented these optimizations in past projects.

Talk about setting up comprehensive logging and monitoring systems using tools like AWS CloudWatch or custom logging in your workflows, ensuring that you capture all critical events, errors, and performance metrics for quick troubleshooting.

Mention resources such as industry blogs, conferences, webinars, and online courses. Discuss any specific communities or forums you engage with to share knowledge and stay informed about advancements in data engineering.

Explain that ETL stands for Extract, Transform, Load, where data is transformed before loading into the target system, while ELT – Extract, Load, Transform – involves loading data into the target system first and then transforming it, suitable for large datasets in cloud storage.

Outline a systematic approach to identify bottlenecks, analyzing areas such as data processing logic, query efficiency, and resource allocation. Suggest techniques like load testing or profiling to gather data and confirm performance improvements.

Discuss compliance with data regulations such as GDPR, implementing encryption, data masking, or access controls to ensure data privacy. Highlight any specific practices you have followed to safeguard sensitive information in your pipelines.

Describe your experience implementing Continuous Integration and Continuous Deployment practices in data engineering. Talk about how these methodologies help automate testing, ensure code quality, and promote collaboration across teams.